Abstract Title Page 

Not included in page count. 



Title: Cluster randomized trial of a large-scale education initiative in the Democratic Republic of 
Congo: Baseline findings and lessons 

Authors and Affiliations: 

John Lawrence Aber, Ph.D. 

Department of Applied Psychology 

Steinhardt School of Culture, Education and Human Development. 

New York University 
Kimball Hall, Room417E 
246 Greene Street 
New York, N.Y. 10003 
Tel: 212-998-5410 
email: lawrence.aber@nyu.edu 

Catalina Torrente, Ph.D. Candidate. 

Department of Applied Psychology 

Steinhardt School of Culture, Education and Human Development. 

New York University. 

Kimball Hall, Room 42 IE 
246 Greene Street 
New York, N.Y. 10003 
Tel: 212-998-5561 
e-mail: cet254@nyu.edu 

Jeannie Annan, Ph.D. 

Research and Evaluation Unit 
International Rescue Committee 
122 East 42nd St. 1 1th floor 
New York, NY 10168 
Tel: (212)551-3100 
e-mail: Jeannie.Annan@rescue.org 



Tom Bundervoet, Ph.D. 

Research and Evaluation Unit 
International Rescue Committee 
122 East 42nd St. 11th floor 
New York, NY 10168 
Tel: (212)551-3100 
e-mail: Tom.Bundervoet@rescue.org 

Anjuli Shivshanker, MPH. 



SREE Spring 2012 Conference Abstract 



Research and Evaluation Unit 

International Rescue Committee 

122 East 42nd St. 11th floor 

New York, NY 10168 

Tel: (212)551-3100 

e-mail: Anjuli.Shivshanker@rescue.org 



SREE Spring 2012 Conference Abstract 




Abstract Body 

L imit 4 pages single-spaced. 



Background / Context: 

Description of prior research and its intellectual context. 

Despite the growing body of domestic research suggesting that school -based 
interventions targeting children's academic achievement and socio-emotional well being can 
result in improved education outcomes and social functioning (Durlak et al., 2011; Alderman, 
Kim, & Orazem, 2003), very few studies have taken place in conflict affected and low-resource 
settings such as the eastern Democratic Republic of Congo (DRC). 

The current study is part of OPEQ (Opportunities for Equitable Access to High Quality 
Basic Education), a 5 -year project aimed at improving teachers' motivation and performance and 
elementary school children's numeracy, literacy and socio-emotional wellbeing in 3 provinces of 
the Democratic Republic of Congo (i.e., Katanga, North Kivu and South Kivu), through 
curriculum development, teacher training and community mobilization. The Cluster Randomized 
Trial of OPEQ presents a unique opportunity to make invaluable contributions to the knowledge 
base in the education sector in DRC and globally. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The main purpose of the current paper is to describe and discuss the scientific and 
practical implications of pursuing rigorous developmental research in a low-income, war- 
afflicted country such as DRC. In addition, the paper aims to explore the individual, household 
and school correlates of children’s academic performance and mental health and socio-emotional 
wellbeing in a large sample of 2 nd to 4 th grade children from Katanga province. Results from 
these analyses will serve to identify key correlates of children’s outcomes, which may be useful 
for relevant stakeholders (i.e., OPEQ implementation team, DRC Ministry of Education) to 
improve their intervention strategies. In addition, estimation of intraclass correlations (ICCs) and 
of the amount of variance explained by the observed co variates will permit a calibration of our 
power analysis for the second wave of data collection. This is particularly important in a context 
in which there is a dearth of background information about children’s academic, and especially, 
socio-emotional outcomes. 

Specifically, we ask: 

How are children’s reading and math performance and socio-emotional well being related 

to: 

1. children’s individual characteristics (i.e., gender, grade, language); 

2. household socio-economic risk and protective factors (i.e., assets, parent 
education, illness); 

3. children’s educational risk and protective factors (i.e., school attendance, lateness, 
distance from home to school) 

4. Teachers’ living conditions (i.e., assets, salary), education level, burnout and 
motivation; and 

5. School and classroom size 
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Setting: 

Description of the research location. 

Data for the current study come from 84 schools spread across 6 educational sub- 
divisions in Katanga, one of the largest provinces of the DRC. DRC has the second lowest 
human development index in the world and has been afflicted by periods of violent conflict for 
the past three decades. Though education continues to expand thanks to community, religious 
and private investments, approximately 31% of school-aged children have never set a foot in 
school. Compared to national averages, children in Katanga and the other provinces served by 
OPEQ are less likely to be enrolled in primary school and are more likely to be orphaned or 
otherwise vulnerable (UNICEF, 2008). 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

Participants were 6,31 1 2 nd (35.2%), 3 rd (33.1%) and 4 th (31.7%) grade children (54.1% 
boys), and 470 teachers (71.1% male; average age = 38) in 84 elementary schools located in six 
educational sub-divisions in the province of Katanga, DRC. Schools were randomly selected to 
participate in data collection for the evaluation from a total of 203 schools grouped in 54 
geographically-defined clusters of 3 to 6 schools. School size ranged widely from 78 to 1,240 
students. According to children’s self-reports, about 14% had gone to bed hungry "often" or 
"sometimes" in the past month, 3.5% reported none of their parents has ever attended school, and 
3.9% reported both of their parents are illiterate. An approximate 87.6% of children reported 
Kiswahili is spoken at home, and only 2.69% reported French, the official language of 
instruction in 3 grade and onwards, is spoken at home. About 6% of children reported being 
"often" or "always" absent from school and arriving late to school. 

The schools, children and teachers are part of an ongoing, longitudinal cluster- 
randomized evaluation of OPEQ. In this paper we explore the correlates of children’s academic 
performance, mental health and wellbeing at baseline of this evaluation. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

The school-based cluster -randomized component of OPEQ consists on teacher 
professional development on a new reading, math and social-emotional learning curriculum. 
Teacher training is delivered by a group of 70 Master Trainers (MT) (one per cluster of 3 to 6 
schools) composed of teachers, headmasters, pedagogical advisors, inspectors and key technical 
staff from the Ministry of Education. MTs are selected based on their successful completion of 
Training of Trainers workshops where they demonstrate competency in both knowledge content 
and delivery of in-service teacher training. MTs participate in the development of the teacher 
training materials and are responsible for carrying out cluster-based workshops for teachers and 
school principals. 

During training, teachers are provided with a core package of print and interactive 
computer-based materials on three key content areas: howto teach reading and math, how to 
ensure student wellbeing and howto assess students. Materials on each content area will be 
provided for each grade level (i.e., 1-6) and integrate pedagogical techniques such as active 
learning methods, use of structured lesson plans and positive discipline within the content areas. 
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Materials on how to ensure student wellbeing instruct teachers on how to integrate the teaching 
of social and emotional skills into their reading and math classes. 

Research Design: 

Description of the research design. 

As mentioned above, a total of 203 schools, nested in 54 geographically defined clusters, 
were selected to participate in the OPEQ intervention in Katanga. The 54 clusters were randomly 
split into three groups of 20, 17 and 17 clusters. This year (2011 -12), OPEQ will be piloted in the 
first group of 20 clusters. The remaining two groups of 17 clusters will receive the intervention 
in either 2012-13 or 2013-14, respectively. Thus, the impact evaluation study in 2012-13 uses an 
experimental wait-list control group design with 20 treatment and 17 control clusters. In 2013-14 
(the second cohort intervention trial), OPEQ will be implemented in both the previous year’s 
wait-list control group clusters and the previous year’s experimental condition for a second year 
(34 clusters). 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

Out of the 203 schools targeted by OPEQ in Katanga, eighty-four representative schools 
were randomly selected for data collection as part of the impact evaluation study. Baseline data 
were collected from a randomly selected sample of 6,3 1 1 children in 2 nd to 4 th grades, and 470 
teachers in grades 1 to 6. Children sampling was stratified within schools and grade levels. 
Sampling of teachers was conducted within schools. 

Assent and consent were requested from all children and teachers at the time of data 
collection and refusal to participate was very rare. Parental consent for children's participation 
was not obtained because parents live very far from the schools and it would have been very 
difficult to obtain written consent from a representative sample of parents. Also, it is not 
common practice in DRC to obtain written consents for children to participate in assessments at 
school and children are accustomed to taking tests and answering questions without parental 
consent. The Ministry of Education and field team widely advertised the evaluation in each 
school and community to ensure that parents were frilly informed and had the opportunity to ask 
any questions, raise any concerns and opt out. 

Students were asked questions about their background and their school and home 
experiences, and were assessed on their math and reading skills and socio -emotional wellbeing. 
Students were randomized to complete different sets of assessments (e.g., math and reading or 
math and socio-emotional well being) with the aim of reducing participant burden. The 
maximum slotted time per child was 45 minutes. 

Teachers were asked to answer a survey about themselves, including their educational 
background, livelihoods, job satisfaction, burnout, motivation and teaching experiences. The 
maximum time slotted for teacher interviews was 90 minutes. 

Teacher and child interviews and assessments were conducted in French (the official 
language) and Kiswahili (the most common local language) by local staff trained in data 
collection procedures by the OPEQ team. 

To accommodate the nested nature of the design, estimates of associations between child 
outcomes and child, household and school characteristics will be calculated using a series of 3 - 
level hierarchical linear models with sub-division fixed effects in HLM 6.02. In these models, 



SREE Spring 2012 Conference Abstract Template B-3 




Level 1 will represent children, Level 2 will represent schools, and Level 3 will represent sub- 
divisions. 



Findings / Results: 

Description of the main findings with specific details. 

Descriptive statistics for key children’s demographic characteristics and reading and math 
performance are summarized in tables 1 through 3. Students were randomized to complete 
different sets of assessments; therefore, the effective sample for which descriptives are presented 
consists of 4,965 children who completed the demographic survey and were administered the 
Early Grade Reading Assessment (EGRA; RTI, 2009) or Early Grade Math Assessment (EGMA, 
RTI, 2010). Descriptives for the full sample will be presented at the time of the conference. 

Overall, our preliminary findings indicate very low levels of performance in math and 
reading, better performance on math compared to reading, higher scores for older compared to 
younger children and for boys compared to girls. Analysis examining the associations between 
children’s academic and socio-emotional outcomes and children’s individual, household and 
school characteristics will be complete by January 2012. 

(Please insert Tables here) 



Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

Descriptive results to date not only reveal very low levels of performance in both math and 
reading in our sample of 2 nd to 4 th grade children in Katanga province, but also hint at important 
gender disparities that will be further explored at the time of the conference. In addition, the fact 
that reading scores appear to be lower than math scores, coupled with the low percentage of 
children who reported speaking French at home, suggests that learning of French as a second 
language may be one of the critical challenges faced by Congolese children in schools. Practical 
strategies to support language learning could include earlier, more extensive and more 
intentional immersion in French, or more instruction (i.e., beyond third grade and/or with more 
supportive materials) in the mother tongue. 

Preliminary results also raise questions about how best to set benchmarks against which 
to measure the success of interventions like OPEQ, in settings in which rigorous evaluations of 
interventions of this nature are rare or poorly documented. For example, OPEQ proposed to 
improve children’s math and reading scores by 30%. However, considering what children gain 
during the primary school years in the absence of interventions, an increase of 30% seems rather 
ambitious. Alternative ways of operationalizing and setting benchmarks to assess the success of 
programs like OPEQ, as well as strategies to increase the probability of producing and detecting 
substantive changes in children’s outcomes, will be discussed at the conference. 

Finally, in addition to the presence and strength of what appear to be important 
differences in performance by grade, gender and geographic location, there may be large within - 
group differences to be described and understood. Identifying other sources of variation in 
children's performance, as proposed in the current paper, is a critical first step in fine-tuning and 
strengthening the strategies to improve children’s outcomes. 
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Appendices 

Not included in page count. 
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Appendix B. Tables and Figures 

Not included in page count. 

Table 1. Percentages of boys and girls per grade and subdivision 



Grade 


2nd 




3rd 




4th 




Subdivision 


% Boys 


% Girls 


% Boys 


% Girls 


% Boys 


% Girls 


Kambove 


55.06 


44.94 


50.60 


49.40 


58.33 


41.67 


Kasenga 


55.92 


44.08 


55.56 


44.44 


63.98 


36.02 


Kalemie 


50.00 


50.00 


47.69 


52.31 


53.17 


46.83 


Kongolo 


50.92 


49.08 


59.33 


40.67 


58.75 


41.25 


Mutshatsha 


50.48 


49.52 


51.99 


48.01 


59.72 


40.28 


Lubudi 


50.27 


49.73 


54.02 


45.98 


51.00 


49.00 


Average across all 
subdivisions 


52.11 


47.89 


53.20 


46.80 


57.49 


42.51 



Table 2. Percentage of children who speak each language by subdivision 1 



Language 

Subdivision 


Lingala 


Kiswahili 


Tshiluba 


French 


Other 


Kambove 


0.20 


95.13 


0.78 


1.95 


33.72 


Kasenga 


0.00 


39.51 


0.00 


3.40 


87.04 


Kalemie 


1.90 


99.53 


0.35 


0.47 


14.87 


Kongolo 


0.20 


98.45 


0.72 


0.52 


63.12 


Mutshatsha 


1.10 


99.23 


2.20 


4.30 


43.39 


Lubudi 


1.00 


93.48 


2.79 


5.49 


47.67 


Average across all 
subdivisions 


0.73 


87.55 


1.14 


2.69 


48.30 



1 Note: percentages do not add up to 100 because some children speak more than 1 language and are thus 
represented in more than one language category. 
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Table 3. Reading and Math scores by gender, grade and subdivision 



Grade 



Gender 



Katanga Subdivisions*** 



EGRA 




2 


3 


4 


Girls 


Boys 


1 


2 


3 


4 


5 


6 




n 


Mean 






Mean 




Means 












Vocabulary (20) 


2939 


6.56 


8.04 


9.58 


7.69 


8.29 


7.82 


5.69 


8.91 


8.57 


8.31 


8.10 


Initial Sound Identification (10) 


2951 


0.68 


1.04 


1.32 


0.89 


1.10 


0.15 


0.06 


2.90 


1.66 


0.33 


0.49 


Knowledge of graphemes (100) 


2205 


11.30 


16.76 


22.99 


16.20 


18.60 


10.87 


8.19 


24.95 


20.41 


20.06 


13.62 


Familiar word reading (50)* 


1900 


- 


3.44 


6.99 


4.22 


5.96 


2.47 


1.72 


10.95 


6.52 


5.21 


2.64 


Invented word decoding (50)* 


1906 


- 


2.78 


5.78 


3.45 


4.91 


1.66 


1.04 


9.39 


5.02 


4.58 


2.28 


Oral passage reading (50)* 


1898 


- 


3.98 


8.75 


4.79 


7.53 


3.02 


1.91 


12.98 


8.95 


6.21 


2.75 


Reading comprehension (5)* 


1919 


- 


0.16 


0.39 


0.22 


0.32 


0.07 


0.04 


0.65 


0.48 


0.19 


0.07 


Listening Comprehension (5) 
Writing a complete sentence 


2951 


0.45 


0.63 


0.90 


0.61 


0.69 


0.28 


0.31 


1.03 


1.02 


0.53 


0.53 


(3)** 


940 


- 


- 


0.81 


0.65 


0.92 


0.63 


0.58 


1.44 


0.67 


0.83 


0.60 








Grade 




Gender 




Katanga Subdivisions* 


** 




EGMA 




2 


3 


4 


Girls 


Boys 


1 


2 


3 


4 


5 


6 




n 


Mean 


Mean 


Mean 


Mean 


Mean 


Mean 












Number Identification (30) 


2920 


14.19 


12.15 


15.72 


13.54 


14.40 


12.16 


10.23 


18.01 


13.26 


16.05 


12.97 


Quantity Discrimination (10) 


2928 


6.29 


5.56 


6.43 


5.98 


6.20 


6.73 


5.01 


6.43 


5.34 


6.54 


6.53 


Missing Number ( 1 0) 


2928 


2.40 


3.38 


4.07 


3.06 


3.42 


2.66 


2.04 


4.54 


3.04 


3.69 


3.14 


Addition (24) 


2935 


4.82 


7.49 


10.77 


7.09 


7.99 


6.75 


4.95 


8.55 


8.71 


8.94 


6.60 


Subtraction (24) 


2935 


3.30 


5.03 


7.66 


4.73 


5.67 


3.54 


2.96 


5.87 


6.83 


6.86 


4.09 


Multiplication (10) 


1900 


- 


0.69 


1.38 


0.77 


1.24 


0.56 


0.84 


1.13 


1.47 


1.10 


0.82 


Word Problems (6)* 


2923 


2.02 


2.35 


3.06 


2.33 


2.57 


2.23 


1.29 


3.07 


2.04 


3.31 


2.48 


Geometric Shape Identification 
(4) 


2936 


2.22 


2.53 


2.72 


2.41 


2.53 


2.65 


1.56 


2.74 


2.53 


2.65 


2.57 


Geometric Shape Naming (4) 


2936 


1.24 


1.58 


1.79 


1.45 


1.59 


1.34 


1.25 


2.37 


1.39 


1.50 


1.27 



* Sub-test not administered to 2 nd graders, * * Sub-test not administered to 2 nd or 3 rd graders, *** 1: Kambove, 2: Kasenga, 3: Kalemie, 4: 
Kongo lo, 5: Mutshatsha, 6: Lubudi 
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